Type of dataset

• Supervised: for each input in the dataset, the expected output is also part of the dataset
• Unsupervised: for each input in the dataset, the expected output is not part of the dataset
• Semi-supervised: only a portion of the inputs of the dataset have their expected output in the dataset
• Reinforcement: there is no predefined dataset, but an environment giving feedback to the model when it takes actions

Type of output

• Classification: assigning one (or multiple) label(s) chosen from a given list of classes to each element of the input
• Regression: assigning one (or multiple) value(s) chosen from a continuous set of values
• Clustering: create categories by grouping together similar inputs

Dataset

Content

Subsets

Neural Network

Deep Learning

Basic Elements

Linear Functions

Activation Functions - Definition

Activation functions allow to approximate much more complex functions, using a sequence of intertwined affine layers and activation layers.

Activation Functions - Examples

Fully Connected

The most basic layer, in which each output is a linear combination of each input (before the activation layer)

Convolutional

A layer combining geographically close features, used a lot to process rasters.

Recurrent

Type of layers designed to process sequential data such as text, time series data, speech or audio. Works by combining input data and the state of the previous time step.

The two main variants of recurrent layers are:

Nowadays, transformer architectures are however preferred to process sequential data.

Pooling

A type of layers used to reduce the number of features by merging multiple features into one. There are multiple kinds of pooling layers, the most simple ones being Maximum Pooling and Average Pooling.

Max Pooling Example^[10]

Residual

A Residual Block aims at stabilizing training and convergence of deep neural networks (with a large number of layers), by adding the input of a given layer to the output of another layer further down in the architecture.

Attention

Attention aims at determining relative importance of each part of the input to make better predictions. It is used a lot in natural language processing (NLP) and image processing.

Attention mechanism in seq2seq with RNN^[12]

And a lot more…

• Dropout: randomly drop out some of the nodes during training to reduce overfitting
• Batch Normalization: normalize the input of each layer across the batch to improve training stability and speed
• Layer Normalization: normalize the input of each layer across the features to improve training stability and speed
• Embedding: transforms discrete input data into continuous vectors with lower-dimensional space
• Flatten: convert multi-dimensional data into 1D data that can be fed into fully connected layers
• …

Fully Connected Network (FCN)

Fully connected network^[13]

Convolutional Neural Network (CNN)

Standard for image processing, with 2D convolutional layers followed by fully connected layers.

1D convolutional neural network^[14]

A Lot More

• Recurrent Neural Network (RNN)
• Generative Adversarial Network (GAN)
• Autoencoder Network
• Transformer Network

Model

Example of fully connected network made with NN-SVG

PyTorch

import torch
import torch.nn as nn

# Simple fully connected model with 2 hidden layers
class SimpleMLP(nn.Module):
    def __init__(self):
        super(SimpleMLP, self).__init__()
        self.fc1 = nn.Linear(2, 20)   # Input layer to 1st hidden layer
        self.fc2 = nn.Linear(20, 10)  # 1st hidden layer to 2nd hidden layer
        self.fc3 = nn.Linear(10, 1)   # 2nd hidden layer to output layer

    def forward(self, x):
        x = torch.relu(self.fc1(x))   # ReLU activation after first layer
        x = torch.relu(self.fc2(x))   # ReLU activation after second layer
        x = self.fc3(x)               # Output layer (no activation for regression tasks)
        return x

Tensorflow + Keras

import tensorflow as tf
from tensorflow.keras import layers, Model

# Simple fully connected model with 2 hidden layers
class SimpleMLP(Model):
    def __init__(self):
        super(SimpleMLP, self).__init__()
        self.fc1 = layers.Dense(20, activation='relu')  # Input layer to 1st hidden layer
        self.fc2 = layers.Dense(10, activation='relu')  # 1st hidden layer to 2nd hidden layer
        self.fc3 = layers.Dense(1)  # 2nd hidden layer to output layer (no activation for regression)

    def call(self, inputs):
        x = self.fc1(inputs)
        x = self.fc2(x)
        return self.fc3(x)

Why Optimization?

The number of possible combinations of parameters is huge, even with small NN models. To (hopefully) find the best possible combination, we need two things:

• A way to evaluate any combination
• A way to find well-performing combinations

Definition

A loss function is a mathematical function that quantifies the difference between the network’s predicted output and the actual target values. The goal during training is to minimize this loss by adjusting the model’s weights, using gradient descent.

The most common loss functions are:

• Mean Squares Error (MSE) for regression
• Cross-entropy Loss for classification

Differentiable

To be able to perform gradient descent, the loss function must be differentiable, which means continuous (no jump) and smooth (no sudden change of direction).

Convex

To get the best results when performing gradient descent, it is also better if the function is convex. The simplest definition of convexity is that if you trace a straight line between two points on the curve, the curve will be below the segment between the two points.

Example of convex and non-convex functions^[15]

Definition

Gradient Descent works because at any point in the definition space of the function, the gradient points in the direction of the steepest angle. So locally, following this direction is the quickest way to get to the lower value of the function. If we come back to the requirements listed before:

• Differentiable functions are functions where the gradient exists everywhere
• Convex functions are convenient for gradient descent because they have only one minimum value and slowly going down the function will always lead to the minimum value.

Algorithm

Gradient Descent boils down to iteratively:

1. Compute the gradient of the loss function at the current point
2. Make a step towards the direction of the gradient to a new point
3. Repeat step 1 until we stop

In this process, the three things that have to be defined are:

• The starting point (weights initialization)
• The size of the steps (learning rate)
• The condition to stop

Weights initialization

The starting point is defined by the first output of the model, and therefore by the initial values of the weights of the model. There are numerous methods to initialize the weights, but the most common one is to randomly initialize them using a centered and normalize Gaussian distribution.

Learning rate

The gradient gives us a direction and a norm, but this norm is arbitrary and has to be rescaled using what we call the learning rate. The learning rate doesn’t define the size of the steps, but the scalar factor to apply to the gradient’s norm, which means that the norm still plays a crucial role.

The choice of the learning rate is crucial to hopefully converge quickly to the global minimum loss.

Example of gradient descent on the same function with different learning rates^[15]

Stop condition

The stop condition determines when you decide to stop the algorithm. An easy solution is to choose a number of steps before launching the algorithm, but this will either imply useless computations after the algorithm has reached a final point, or stopping too early and not get the best results possible.

Therefore, although there are more complex methods, the most common and simple process is to monitor the value of the loss, memorize the lowest value ever reached, and stop when there has been a given number of steps without any improvement to the best value. Then, we usually keep the model weights corresponding to this best value.

Unlucky examples

Why?

Gradient Descent is beautiful, but right now, we only know in which direction (the gradient) the output of the model should go. To transmit this information to the weights of the layers of the model, we use backpropagation.

Definition

How?

Backpropagation involves a lot of computations of partial derivatives, which are individually not difficult but very bothersome. Happily, NN libraries handle backpropagation automatically by calling only one function, so no need to worry about it.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Generate some simple data
X = torch.randn(100, 2)  # 100 samples, 2 features
y = (2 * X[:, 0]**3 + 0.5 * X[:, 1]**2 + torch.randn(100) * 0.5).unsqueeze(1)

# Create a DataLoader for batching
batch_size = 32
dataloader = DataLoader(TensorDataset(X, y), batch_size=batch_size, shuffle=True)

# Instantiate the model
model = SimpleMLP()  # A simple model that is assumed to be defined before

# Define a loss function and optimizer
criterion = nn.MSELoss()  # Mean Squared Error Loss
optimizer = optim.SGD(model.parameters(), lr=0.01)  # Stochastic Gradient Descent

# Training loop
epochs = 1000
epoch_losses = []  # List to store average loss at each epoch

for epoch in range(epochs):
    epoch_loss = 0.0  # Accumulate loss for this epoch
    for batch_X, batch_y in dataloader:
        # Forward pass
        predictions = model(batch_X)  # Compute the model's predictions
        batch_loss = criterion(predictions, batch_y)  # Compare predictions to true values
        
        # Backward pass and optimization
        optimizer.zero_grad()  # Reset gradient
        batch_loss.backward()  # Backward pass
        optimizer.step()  # Update parameters

        # Accumulate weighted loss for the batch
        epoch_loss += batch_loss.item() * batch_X.size(0)
    
    epoch_loss /= len(dataloader.dataset)  # Normalize by dataset size to get average epoch loss
    epoch_losses.append(epoch_loss)  # Store the epoch's total loss

    # Print loss every 10 epochs
    if (epoch + 1) % 100 == 0:
        print(f"Epoch {epoch + 1:>4}/{epochs}, Loss: {epoch_loss:.4f}")

Tensorflow + Keras

import tensorflow as tf
from tensorflow.keras import layers, Model
import numpy as np

# Generate some simple data
X = np.random.randn(100, 2).astype(np.float32)  # 100 samples, 2 features
y = (2 * X[:, 0]**3 + 0.5 * X[:, 1]**2 + np.random.randn(100) * 0.5).astype(np.float32)
y = y.reshape(-1, 1)  # Reshape y to (100, 1)

# Create a Dataset for batching
batch_size = 32
dataset = tf.data.Dataset.from_tensor_slices((X, y)).batch(batch_size)

# Instantiate the model
model = SimpleMLP()  # A simple model that is assumed to be defined before

# Define a loss function and optimizer
loss_fn = tf.keras.losses.MeanSquaredError()  # Mean Squared Error Loss
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)  # Stochastic Gradient Descent

# Training loop
epochs = 1000
epoch_losses = []  # List to store average loss at each epoch

for epoch in range(epochs):
    epoch_loss = 0.0  # Accumulate loss for this epoch
    for batch_X, batch_y in dataset:
        # Forward pass
        with tf.GradientTape() as tape:
            predictions = model(batch_X)  # Compute the model's predictions
            batch_loss = loss_fn(batch_y, predictions)  # Compare predictions to true values
        
        # Backward pass and optimization
        gradients = tape.gradient(loss, model.trainable_variables)  # Backward pass
        optimizer.apply_gradients(zip(gradients, model.trainable_variables))  # Update parameters

        # Accumulate weighted loss for the batch
        epoch_loss += batch_loss.numpy() * len(batch_X)
    
    epoch_loss /= len(X)  # Normalize by dataset size to get average epoch loss
    epoch_losses.append(epoch_loss)  # Store the epoch's total loss

    # Print loss every 10 epochs
    if (epoch + 1) % 100 == 0:
        print(f"Epoch {epoch + 1:>4}/{epochs}, Loss: {epoch_loss:.4f}")

Principle

Reasons

• Limited amount of data for the new task
• New task is similar to old task
• Training a model is very costly
• Better final performance with less overfitting

Applications

Two major applications:

• Computer Vision
• Natural Language Processing

Fine-tuning

Feature Extraction

Multitask Learning

Knowledge Distillation

PyTorch

• Website:
• Flexibility, ease of debugging, beginner-friendly
• Favored in academia and by researchers

TensorFlow

• Website:
• Scalable, production-ready and easy deployment
• Favored in production and industry

Keras

• Website:
• User-friendly, high-level and powerful with TensorFlow integration
• Favored for first hands-on experience for beginners and for prototyping

Playground

Playground

YOLOv8 Model